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METHOD AND APPARATUS FOR As the size of the network increases, the amount of control 

PROVIDING CONTROL INFORMATION IN A traffic required to process the queries increases correspond- 

SYSTEM USING DISTRIBUTED ingly. 

COMMUNICATION ROUTING Therefore, a need exists for a method and apparatus that 

; k I h li i i In i ii ill 

RELATED APPLICATION forwarding packets within a distributed router. 

^!!ff™^ i f^ a ™ t '; ' 11 h " n BRIEF DESCRIPTION OF THE DRAWINGS 

entitled "METHOD AND APPARATUS FOR PROVIDING 

DISTRIBUTED COMMUNICATION ROUTING" that has FIG. 1 illustrates a block diagram of a data network in 

er of Ser. No. 09/352,563 and which accordance with the present invention; 
, 1999, the same day as the present FIG. 2 illustrate graph* ntation of a technique 

for providing forwarding information within a distributed 

TECHNICAL FIELD OF THE INVENTION FIG. 3 illustrat , graphi Ire] entation of 'an alternate 

l i I i ] I i il 

ers within such data networks. FIG. 4 illustrates a block diagram of multiple distributed 

routers in accordance with the present invention; 
BACKGROUND OF THE INVENTION 20 nG s illustralos a llow diagram of a meth od for provid- 

Data networks that utilize an IETF communication pro- m S distributed communication routing in accordance with 

locol .1 mdnrd include , plin ilit> of routers that ire interop- lhe P Tesent invention; 

i 1 1 I 11 i 11 II I R j 6 ill 

devices, such as personal computers, video telephones, „ 5 for providing distributed co. 
telephones. ]iic-.imil^ machines, etc., are coupled to the n;' ' '' ' 
network segments and transfer data to each other via the FI 

s. Iii ordei to uppori uel 1 1 n n pori ition the forwarding engine to facilitate distributed co 
1 accordance with 1 ho present in\onl: 



protocols. Generally, the present in 

To further support ilk t Mi 1 r liter include apparatus that includes proc 

forwarding engines. The route computation engine executes ing registration irifornfation from a pluralfr/of forwarding 

llie muling proloeo I I 1 I 

1 r 1 1 1 11 base (FIB) ) The forwarding engines forward m ti n 1 1 11 1 l j I I 

packets from one network segment to another based on 40 at least one specific forwarding table for at least one corre- 

u ill 1 I in ill I 1 j 1 1 1 plumli 

In legacy routers (i.e.. routers thai 1 1 u till hi I ill i i rdi 

networks for some time), the software and hardware of the generated for each of the plurality of forwarding engines. 

' — -nc and of the forwarding engines are contained i.e.. each forward in bi engine has its own lorwarding table. 

in me same physical structure. As such, the interaction 45 The processing then continues by forwarding the specific 

control of data transportations between the routing ** t 
ne and forwarding engines was trivial. 




engines. The processing then continues 
of the query, the routing engine, or route least one specific internal forwarding ta 
I I I I 11 fh 

other local databases, and responds vulh the appropnale to other forwarding engines within the distributed router. 

Ill It lc re forwarded to the corresponding for- 

wilh multiple accessible networks and hosts, a substantial warding engine or engines. With such a method and 
amount of communication between the forwarding engines apparams. more ellicient distributed forwarding mformation 
and the route compulation engine is required just to handle is obtained, thus saving significant computational overhead 
the query response lechanism. h 1 I tune a 65 within a distributed router. 

piery the rou 1 brain 1 1 formation 1 the pi i r I _i 1 I 1 fbi 
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iii I i _ i u i i i ii of the TCP/IP or U] 

! i| i iter. In one embodi i i I | ii In l l 

lion engine can pass messages through one or more ol' Ihc shared by multiple forwarding engines il the underlying 

forwarding engines lo external roulers ooupled lo Ihe lor- connection technique suppotas such sharing. VYhen a number 
ii 1 1 [ i [ e 1 ills iter peer f of different traffic flows are available between the route 

protocols. The route eompul i nengin , in Ise send slants computation engine 38 and the plurality of forwarding 

queries to the forw aiding engines and update the status of engines 40-48, different levels, or qualities, of service may 

the distributed router based on the responses. The route be provided over the different traffic flows. Thus, the quality 

computation engine can also pro\ide additional control ol ei i s isi ' il n Ii i ih t in compulalion 
information to one or more of the forwarding engines LO engine 38 to one forwarding engine may be greater than the 

rej lie, format! ag oi other data presentation or encapsu- quality of service on the traffic flow to another forwarding 

lalion requirement of receiving endues lo which Ihc lor- engine. Nolo lhal a comhin.ilion sealing and separauon any 

i ii i , ling data In addition to these be utilized to provide various forwarding engine groupings 

capabilities, varying levels of service may be provided with different control traffic qualities of service. 

1 1 I n ' i' 1 " 1 Ii 2 Hi trates a graphical representation of ihc logi- 

" 1 i i, I i i i i iii, i a hi I n 

can be segregated in terms of quality of service. tables 50-S8. The connections between the forwarding 

The presenl invenlion can be more fully described with engine 42 and. its neighboring roulers and forwarding 

I l i I 111 12 36 interoperabh of the forwarding table generation I c 1 ric II li il 

t 411 4S 1 I 

i ' i i i ii ill l iralion inlormal'ion 611 includes Ihe iden I 

Router 22 is a logically distributed router lhal includes a ing engine and „[Im luiKl, ,n ,1 ml ,n i ,n , pi lb 

I I I 4 I 4> I ^ l" L '| | l' 1 ' " 4 

i n, nl, =l, ill icn i ?2 1 be pi ing 1 " 1 Hers arc included in Ihc ri n lormali 

module 50 may be single processing device or a plurality of Upon reccn ing Hie registration information 60, the route 

processing devices. Such a processing device may be a ampul n n engine 3 i i i 1 n nl rm i n 

microprocessor, microcontroller, digital signal processor, base 62 that is provided to the forwarding engines. The 



i I ii I i i ~ I 

llelle P I II _lle 1 1 I u 1 4 ' ' ' ' " 

ded within Ihe circuit uprising 1 late niachin I " 1)1-15 

logic circuitry. The operational instructions stored in D2-I5 

memory 52 and executed by processing module 50 will be D3-I4 

or detail with reference to ihe processing |)4-B 

' the graphical w ]5 js ^ identity given t0 connecnon or coup ling 
belv.een forwarding engine 42 and router 16. 14 is Ihe 

The forwarding engines 40-48 each include a processing identity given to connection oi coupling between forwarding 

module 54 and memory 56. The processing module 54 may erjgine 42 a„d router 18. 13 is the identity given lo ™-. 

be of a similar type i u n i, hi *l> and memory 56 tionoree plii II engine 42 and 1_. 

I ii i i | lemeiy 52. In general, lb , 44 , nu cue. ine 42 slil nt 

mpulalion engine 38. which refer, 1 I I 1 11 ,1 n I 

I I li I i [ 1 i i , 1 1 

' s l, , ,, , H llte I liim engine. The notation 1)1-15 i use 1 lo inrii- 

tributed router 22. The information a, compiled to produce cate t0 the forwarding engine diet all data communication 

-'il' i- i n 1 hi ro\ ided to each of the packets to destination I )1 si,,, uld be forwarded via connec- 

forwarding engines 40^48. The forwarding engines 40^t8 tion or coupling 15. 

utilize the specific forwarding tables to route data to other Once Ihe registration process has been completed, the 

routers 12 20 and 24 -36. r ,„„ c computation processor 38 may send status request or 

The interconnections, or traffic flows, between the route "hello" messages to one or more of the forwarding engines 

ii 3 11 i « ' S 4 i 4 V i t til I i I i l 
' y be of the maintenance of the distributed routing system. I he 

rote. status requ t i nl onlrol i , mation that may be 




Therefore when the e 1 ? o* tk title m be 1 pi ihty of forwarding 





ihtnimlion din fines . , . 

mges may be the en g me as either specific changes 



s tunneling messages to 



FIG. 7 m 



Thepr 

forwarding table is updated 

' " '" m changes 

' ' to the distributed 
11 i i i i ill eon ih ii 

11 ii n or. Hi i In Hi i ! i I 

links, and/or any change that would cause a need for 
updating ihe Ibrwarding lnlbiTnaiion. These changes may be 

delected I ]i in i. lorwa in<_ on im I >i Villus tionmay indicate a level of service to be provided To apacket 
or may indicate particular header information, data 
encapsulation, etc. that is appropriate for packets forwarded 

^P-eeds,,, , , MU , mlex.e, ^.X^aStSSt: 

precuile. oilier embodiments 



upd-iies ihm 
described col 



wilh reference lo 1'ICi. 3. I he process Ihcn proceeds [o slop 
106 where at least one external fo u i i. 1 1 1 I i - i 

, ] t, ii 11 1 , v 'ii i i _ I I i 1 e 
generated for all of the forwarding or a corresponding 
st< nal i rdingta 1- maj be generated for grouping 



without deviating Iron 



providing distributing cc 



lc forwarding er 



routine. Ihe method comprises Ihe slops ot: 
■ c p^e^Sproceeds to step 108 where at least one 35 a > obtaining at a route computation engine registration 
ilk mi in il I ,1, nil , . „ , lli ii 1 isi 1111 1111111 " " 111 ' t 1,11 ,l " 1 1 1 1,1111 
res] i I i i i i v il In i I I . 
leas | i t i I l , I ' I islral ion i li n I 
internal forw u Ii i I nlm .In Hi irding <0 providing control information to at least one forward- 
engines, a corresp. -nJiut internal forwarding table for each 40 m engine of the plurality of forwarding engines for at 
of the plur lih I I i corre ponding lea t one t I i lenance and a 
i I i ii I I i ping of the plurality specific data forwarding operation. 
In mi iii I p hen process t lop - Hit moll. «l of cl lim I a Herein pn i lin « tr I 
110 where the internal and external forwarding tables are information further comprises pro\ lding control inf 



5 for distributed re 

information includes a control m 



updated based on fication of present functionality of the at least one forward- 
conliguralton changes el the nelv.ork. ing engine. 

The methods of claims 5 and 6 may both include the route 
compulation engine perl rmii i 1 functions for a second ^ 
plurality of forwarding engines. As such, a single compu- 
tational engine may perform like functions for multiple 
distributed routers. Tins was described wilh reference to 
FIG. 4. These methods may also include the route compu- 
tation engine peri rmi i Mi il i figuration of the 5< 
various forwarding engines using control messages. In 
addition, once the forwarding engines have registered and 

indicated llieir interlace connections, tunneling messages present functionality further comprises 
may be sent to and from the route computation engine via present fun. i nt i ed mi at least one . 
the forwarding engines. 60 from the at least one forwarding engine, ; 

FIG. 7 illustrates a logic diagram of a method for a response from the at least one forwarding er 
forwarding engine to facilitate distributed cc .... 
routing. The process begins at step 120 where a forwarding 
i i i k iii lilt I I i i 

proceeds to step 122 where ai least one forwarding table is 
received in resp' I 
forwarding table may be a speuht. i rding tab] Qirol inl ition 1 in the control information. 
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7. The method of claim 1, wherein providing control processor via a plum lit} ol connections, wherein the plural- 
.n forma lion lurlher comprises providing the conirol in lor- iiy of connections provide a plurality of levels of sen ice for 
1 11 ! 1 Ll1 1 ! 1 hu n the providing control information to the plurality of forwarding 



dala p;.ickel. 



I i .e tng module; and 



rk routing element comprises-: 



8. The method of claim ?. v. herein me control message is memory operably coupled tc 

i i i 1 1 i nt i the al leasl one Jala wherein the memory stores t. 

p nil! e Hi unci n [a la in n ! i can i 1 roeessin i i 

leasl one router packet directed to at leasl one router .annul i a cut; nc i isir 
operably coupled to the at least > 



information for distributed routing maintenanci 



the at least one router, wherein the 

includes information describing i; 

I ] i iii i li II i i ii i 

9 ["hem thod i laiml v herein the control information 
further comprises packet conjuration lnlormaiion. w herein 

if c e I i ml mi ition determines packet for- 2) ; 
mats of packets exchanged between forwarding engine 

Ml. The method of claim i. wherein providing cor 

' i i 1 1 1 I i I I L i I I k III 

ii li t i I ! II I 1 1 1 t tic I leasl 

jl I 1 I 1 j 1 I || 1 | 1 k I I 19 

' ' ' | |'l | [ 1 | l' ! " K " n 1 1 i" 11 I I I 111 I I I 

ill n i n la i i I i iii i a i eating a stale of each interlace associated w iih the al least 

11. The method of claim 1, wherein providing control one forwarding engine, 

information tiirth r compri i idi the control infor- 30 21. The distributed network routing element of claim 18, 

''onto ip i i fiheplurali i rdi nginesvia whereii the memory further comprises operational instruc- 

"" " tions thai cause the processing module to: d) determine the 

■alions routing eircuil com- present lu Ii i lea.-;! one lorvvarding engine: 
and e) update registration information of ihe plurality of 

i plurali M 1 i „m L s that are interoperably 35 forwarding engines based on the present functionality. 

coupled to permit data exchange; and 22. The distributed network routing element of claim 21, 

a route computation engine operably coupled to the wherein the memory further comprises operational instruc- 

computation cngim re es regi 'r iti □ ink imation present functionality based on at least one of: a response 

from the pluralitv of forwarding engines, wherein the in from the at least one forwarding engine, and a lack of 

i i i mi neluiic li iiplm infoni n n response from the at least one forwarding engine. 

! ! a i 1 i_ ii ihe plural I 1 irward- 23. The distributed network routing element of claim 18, 

ing engines and a plurality of external routers, wherein wherein the memory further comprises operational instruc- 

malion to al leasl one forwarding engine of ihe plurality 4-" data message, wherein al leasl one data packet is included in 

of forwarding engines for at least one of: distributed the tunneling data message, wherein the at least one data 

muling maintenance an lie Ibi i i ckcl is di I to at 1 i [ K \ 

tion. ihe at leasl one forwarding engine, wherein the a! least one 

13. The circuit of claim 12, wherein the control informa- forwarding engine forwards the at least one router packet to 
lion includes al lea tali > qu I herein (he k the al lea i 1 1 t i Mi i i nlbrmation 
one lerwarding engine provides a status response to the includes in forma lion describing couplings between the plu- 
mule, compulation engine in response lo ihe status request. ralilv of forwarding engines and routers. 

j i i I 24. The disl I routing clem l> 

the at least one forwarding engine. wherein the control information further comprises packet 

14. Ihe eircuil of claim 13. wherein ihe present function- -' v c n in lorn Iiei in if pad i onli u I m 

ality of the forwarding engine includes state of each inter- information determines packet formats of packets 

face associated with the forwarding engine. exchanged between forwarding engines. 

15. The circuit of claim 14. wherein the route computation 25. The distributed nen\ oi k n ailing element of claim 18, 
i regislralion information irdii in the memory li i I mstri 

engine based on the status response. 60 tions mat cause the processing module to pro vide the control 

16. The circi ill i le st p i n nt r i II I that \ I 
the | 1 1 I I 1 engines are grouped into at least lo I plurality of forwarding engines, wherein Ihe plurality 

irding engi t i ipled to ll i i i i ii I i I t i i _ 

hi i ii i i ued traffic flow. 65 wherein each subset includes at least one forwarding engine. 

orwarding engines are cou te route computation ***** 



